A PRACTITIONER'S DECK  ·  ADITYAA S. CHANDRAMOHAN

LiveKit & the QA of
Voice AI Agents

How real-time voice AI is built on LiveKit — and how quality engineering teams validate it with Arize Phoenix and modern AI evaluation platforms.

LIVEKIT AGENTS WEBRTC ARIZE PHOENIX LLM-AS-JUDGE VOICE AI QA CI/CD EVAL GATES
Adityaa S. Chandramohan  ·  Senior Quality Engineering Leader  ·  GenAI QA Strategist
01 · WHAT IS LIVEKIT

Open-source infrastructure for the voice-driven internet

Infrastructure for the voice-driven era of computing.

200K+
Developers
AI labs, robotics companies, and Fortune 500 enterprises build on LiveKit — OpenAI's ChatGPT Voice Mode runs on it.
2
Deployment Models
Fully open-source and self-hostable (Apache-2.0, ~13K GitHub stars), or managed via LiveKit Cloud with a global edge network.
1
Core Job
Own the hard real-time plumbing — WebRTC transport, media routing, turn-taking, interruption handling — so developers write agent logic, not networking code.
02 / 11
02 · ARCHITECTURE

Anatomy of a voice agent

USER
Mic audio via WebRTC into a LiveKit Room
→
STT
Streaming speech-to-text (Deepgram, Whisper…)
→
LLM
Claude/GPT generates the agent response
→
TTS
Text-to-speech (ElevenLabs, Cartesia…)
→
USER
Audio streamed back in real time
AGENTS FRAMEWORK HANDLES
VAD & turn detection (knowing when the user is done speaking)
Barge-in (cancel TTS mid-stream when interrupted)
Tool calls (agent invokes APIs mid-conversation)
Realtime APIs (speech-to-speech models with no STT/TTS hops)
TYPICAL USE CASES
Contact center & IVR replacement (voice bots, intake & triage)
Robotics teleoperation and embodied AI
Live translation, tutoring, and companion apps
03 / 11
03 · THE QA PROBLEM

Why voice AI breaks traditional testing

NON-DETERMINISM
The same utterance produces a different response every run. Fixed-output assertions are useless; quality must be scored, not asserted.
LAYERED FAILURE DOMAINS
A defect can live in WebRTC transport, STT transcription, LLM reasoning, tool calls, TTS rendering, or turn-taking logic. Each layer needs its own oracle.
LATENCY IS A FEATURE
Conversation collapses beyond ~1s of response delay. p95 time-to-first-audio is as much a quality gate as answer correctness.
SEMANTIC CORRECTNESS
"Technically responded" is not "answered correctly." You need relevance, faithfulness, and compliance evaluation at scale.
04 / 11
04 · REFERENCE ARCHITECTURE

Wiring LiveKit into an eval pipeline

LIVEKIT AGENT
Session instrumented with OpenTelemetry
→
TRACES
Spans + full transcripts, per turn
→
ARIZE PHOENIX
Trace store, datasets, experiments
→
LLM-AS-JUDGE
Relevance · hallucination · toxicity · correctness
→
GATES
CI thresholds + human review queue
WHY THIS WORKS
Structured session data — LiveKit Agents emits transcripts and events programmatically, not scraped from logs.
OTel-native ingestion — Phoenix reads OpenInference/OpenTelemetry spans directly; one instrumentation pass serves debugging and evaluation.
Production feeds regression — every conversation becomes a candidate test case, sampled into golden datasets.
05 / 11
05 · TEST STRATEGY

A layer-by-layer validation model

LAYER
WHAT QA VALIDATES
ORACLE & TOOLING
Transport (WebRTC)
Join/reconnect reliability, audio quality under packet loss & jitter
Load-testing CLI, network shaping (tc/netem)
STT
Word error rate — accents, background noise, domain vocabulary
Reference transcripts + WER scoring (jiwer)
LLM / Agent Logic
Relevance, faithfulness, completeness, tool-call correctness
Arize Phoenix evals, LLM-as-judge, golden datasets
TTS
Pronunciation of domain terms, time-to-first-byte
Playback review sampling + latency spans
End-to-End
Turn-taking, barge-in recovery, p95 latency, session success
Simulated callers (scripted TTS personas) + trace analysis
06 / 11
06 · ARIZE PHOENIX

The eval platform, up close

PHOENIX BUILDING BLOCKS
Tracing — every conversation is an inspectable OTel trace
Datasets — golden conversations with expected behaviours
Experiments — replay a dataset against a new build, score every output
Evals — hallucination, relevance, toxicity, correctness
CUSTOM JUDGES FOR VOICE DOMAINS
Compliance judge — PII/PHI leakage, skipped disclosures
Policy judge — do stated answers match the knowledge base?
Escalation judge — did it hand off to a human when it should?
Conversation judge — coherence and appropriate concision per turn
THE QA DISCIPLINE LAYER
Judges are themselves LLMs, so QA must validate the validators: benchmark judge scores against human annotation samples, measure inter-annotator agreement, and recalibrate rubrics on drift. An uncalibrated judge is a second opinion, not an oracle.
07 / 11
07 · SIMULATION TESTING

Synthetic callers as regression suites

PERSONAS
Scripted journeys: happy path, angry, accented, interrupting, noisy
→
TTS CALLER
Synthetic audio speaks into the LiveKit room like a real user
→
AGENT UNDER TEST
Full production pipeline responds
→
JUDGE & GATE
Scored against rubric; CI fails below threshold
SCENARIO COVERAGE THAT MATTERS
Barge-in storms — interrupt every 2-3 seconds, verify recovery
Silence & dead air — does the agent re-prompt or hang?
Adversarial inputs — spoken prompt injection, off-topic pivots
Long sessions — context drift after 20+ turns
PRODUCTION FEEDBACK LOOP
Sample live conversations into Phoenix as new golden candidates
Run human-in-the-loop annotation on the sample — the calibration set
Every escaped defect becomes a new persona script
08 / 11
08 · PLATFORM LANDSCAPE

Arize & the alternatives

The platform varies. The pattern doesn't: traces in → judged scores out → thresholds enforced in CI.

PLATFORM
SWEET SPOT
VOICE-AGENT FIT
Arize Phoenix
OTel-native tracing + experiments + judges, open source
Traces map 1:1 to agent sessions
RAGAS
RAG metrics: faithfulness, context precision/recall
Use when the agent answers from a knowledge base
DeepEval
Pytest-style eval assertions in CI
Developer-friendly for teams that own the agent logic
LangSmith
Tracing + evals inside the LangChain ecosystem
Natural fit if the agent is LangChain/LangGraph-built
Braintrust
Experiment comparison across prompts/models
Model or prompt swaps before a wider rollout
09 / 11
09 · METRICS & RELEASE GATES

Numbers that block a release

LATENCY
p95 time-to-first-audio under 1.5s (end of user speech to agent audio); STT partial under 300ms, TTS first byte under 400ms; zero audio gaps over 2s mid-response.
ACCURACY
Hallucination rate below threshold on the golden dataset; tool-call success at 98%+ with argument correctness judged; WER within budget per accent/noise cohort.
SAFETY & COMPLIANCE
Zero PII/PHI leakage on the redaction test suite; 100% required-disclosure adherence in regulated flows; jailbreak/prompt-injection resistance on the adversarial set.
CONVERSATION QUALITY
Barge-in recovery rate and turn-taking correctness; containment vs. human-escalation correctness; judge scores trending, calibrated against a human sample.
10 / 11
10 · TAKEAWAYS

The rails, the ruler, and the gate

LIVEKIT = THE RAILS
Real-time transport, agent orchestration, and the WebRTC layer voice AI runs on — open source, production-proven at OpenAI scale.
PHOENIX = THE RULER
Traces, datasets, experiments, LLM-as-judge — scored on relevance, faithfulness, safety, and latency, continuously.
QA = THE GATE
Simulation suites, calibrated judges, human-in-the-loop sampling, CI thresholds. No score, no ship.
Voice AI doesn't lower the quality bar — it raises it. Latency, accuracy, safety, and conversational grace must all hold at once.
Adityaa S. Chandramohan  ·  Senior Quality Engineering Leader  ·  GenAI QA Strategist
adityaa-portfolio.pages.dev  ·  linkedin.com/in/adityaa-chandramohan
11 / 11
{{ w.text }}
NARRATION
AUDIO NARRATION
Click to begin the presentation with spoken narration and live captions.